Papers with MCQA datasets
Calibrating the Confidence of Large Language Models by Eliciting Fidelity (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models with RLHF and RLAIF have good alignment but exhibit overconfidence post-alignment. |
| Approach: | They propose a plug-and-play method to estimate the confidence of large language models. |
| Outcome: | The proposed method has shown good calibration performance on 6 RLHF-LMs on four MCQA datasets. |
Clues Before Answers: Generation-Enhanced Multiple-Choice QA (2022.naacl-main)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) uses text-to-text framework . but, there is an under-utilization of the decoder and knowledge that can be decoded . |
| Approach: | They propose a generative multiple-choice question answering model which generates a clue from the question and leverages it to enhance a reader for MCQA. |
| Outcome: | The proposed model outperforms text-to-text models on multiple MCQA datasets. |
Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts . |
| Approach: | They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone . |
| Outcome: | The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain. |
Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing MCQA datasets are small in size, which increases difficulty of model learning and generalization. |
| Approach: | They propose a multi-source meta transfer framework for low-resource multiple-choice question answering . they extend meta learning by incorporating multiple training sources to learn a generalized feature representation across domains . |
| Outcome: | The proposed framework is independent of backbone language models and can bridge the distribution gap between training sources and target. |
Generating Multiple-choice Questions for Medical Question Answering with Distractors and Cue-masking (2024.lrec-main)
Copied to clipboard
| Challenge: | Medical multiple-choice question answering (MCQA) requires high accuracy to be useful in practice. |
| Approach: | They propose to focus masked language modeling on disease name prediction when using medical encyclopedic paragraphs as input. |
| Outcome: | The proposed model outperforms the masked language model on disease name prediction and masks the cues to the answers. |
LLMs (Almost) Never Abstain Under Medical Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Medical multiple-choice question answering (MCQA) benchmarks assume large language models should always commit to an answer. |
| Approach: | They propose a benchmark to evaluate medical abstention under uncertainty . they remove the gold answer and introduce an explicit "I abstain" option . results highlight abstinence as a critical but overlooked dimension of medical decision-making evaluation . |
| Outcome: | The new benchmark evaluates medical abstention under uncertainty. |